GARDE: a standards-based clinical decision support platform for identifying population health management cohorts.

Population health management (PHM) is an important approach to promote wellness and deliver health care to targeted individuals who meet criteria for preventive measures or treatment. A critical component for any PHM program is a data analytics platform that can target those eligible individuals. OBJECTIVE: The aim of this study was to design and implement a scalable standards-based clinical decision support (CDS) approach to identify patient cohorts for PHM and maximize opportunities for multi-site dissemination. MATERIALS AND METHODS: An architecture was established to support bidirectional data exchanges between heterogeneous electronic health record (EHR) data sources, PHM systems, and CDS components. HL7 Fast Healthcare Interoperability Resources and CDS Hooks were used to facilitate interoperability and dissemination. The approach was validated by deploying the platform at multiple sites to identify patients who meet the criteria for genetic evaluation of familial cancer. RESULTS: The Genetic Cancer Risk Detector (GARDE) platform was created and is comprised of four components: (1) an open-source CDS Hooks server for computing patient eligibility for PHM cohorts, (2) an open-source Population Coordinator that processes GARDE requests and communicates results to a PHM system, (3) an EHR Patient Data Repository, and (4) EHR PHM Tools to manage patients and perform outreach functions. Site-specific deployments were performed on onsite virtual machines and cloud-based Amazon Web Services. DISCUSSION: GARDE's component architecture establishes generalizable standards-based methods for computing PHM cohorts. Replicating deployments using one of the established deployment methods requires minimal local customization. Most of the deployment effort was related to obtaining site-specific information technology governance approvals.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Feasibility and acceptability of a game-based symptom-reporting app for children with cancer: perspectives of children and parents.

BACKGROUND: Children with cancer have difficulty identifying and describing the multiple symptoms they experience during hospitalization and between clinical encounters. Mobile health resources, including apps, are potential solutions to support child-centric symptom reporting. This study evaluated the feasibility and acceptability of a newly developed game-based symptom-reporting app for school-age children with cancer. PROCEDURE: Nineteen school-age children (6-12 years of age) receiving treatment for cancer at a COG institution in the Intermountain West of the United States used a game-based symptom-reporting app between clinical visits. Feasibility was evaluated through a summary of actual days of app use and interaction with each of the app's features. Children and their parents participated in interviews regarding the app's acceptability. RESULTS: Children used the app a median of 4 days (range 1-12) and interacted most frequently with the symptom reporting and the drawing features. Children enjoyed aspects of the app that supported their creativity and provided choices. Parents endorsed the interactive nature of the app and the value of the child providing his/her own report. Both children and parents identified additional opportunities to enhance the child's user experience. CONCLUSION: Study results support the preliminary feasibility and acceptability of the app. Children's and parents' responses supported the developmental relevance of the app and its role in enhancing the child's autonomy and serving as an outlet for creativity. Future directions include optimizing the child user's experience and investigating the app's role as a resource to enhance shared decision-making for symptom management.

Developing and Validating a Model for Detecting Longitudinal Inconsistencies in the Electronic Problem List.

Clinicians from different care settings can distort the problem list from conveying a patient's actual health status, affecting quality and patient safety. To measure this effect, a reference standard was built to derive a problem-list based model. Real-world problem lists were used to derive an ideal categorization cutoff score. The model was tested against patient records to categorize problem lists as either having longitudinal inconsistencies or not. The model was able to successfully categorize these events with ~87% accuracy, ~83% sensitivity, and ~89% specificity. This new model can be used to quantify intervention effects, can be reported in problem list studies, and can be used to measure problem list changes based on policy, workflow, or system changes.

Looking Behind the Curtain: Identifying Factors Contributing to Changes on Care Outcomes During a Large Commercial EHR Implementation.

OBJECTIVE: To identify factors contributing to changes on quality, productivity, and safety outcomes during a large commercial electronic health record (EHR) implementation and to guide future research. METHODS: We conducted a mixed-methods study assessing the impact of a commercial EHR implementation. The method consisted of a quantitative longitudinal evaluation followed by qualitative semi-structured, in-depth interviews with clinical employees from the same implementation. Fourteen interviews were recorded and transcribed. Three authors independently coded interview narratives and via consensus identified factors contributing to changes on 15 outcomes of quality, productivity, and safety. RESULTS: We identified 14 factors that potentially affected the outcomes previously monitored. Our findings demonstrate that several factors related to the implementation (e.g., incomplete data migration), partially related (e.g., intentional decrease in volume of work), and not related (e.g., health insurance changes) may affect outcomes in different ways. DISCUSSION: This is the first study to investigate factors contributing to changes on a broad set of quality, productivity, and safety outcomes during an EHR implementation guided by the results of a large longitudinal evaluation. The diversity of factors identified indicates that the need for organizational adaptation to take full advantage of new technologies is as important for health care as it is for other services sectors. CONCLUSIONS: We recommend continuous identification and monitoring of these factors in future evaluations to hopefully increase our understanding of the full impact of health information technology interventions.

Data standards for interoperability of care team information to support care coordination of complex pediatric patients.

OBJECTIVE: Seamless access to information about the individuals and organizations involved in the care of a specific patient ("care teams") is crucial to effective and efficient care coordination. This is especially true for vulnerable and complex patient populations such as pediatric patients with special needs. Despite wide adoption of electronic health records (EHR), current EHR systems do not adequately support the visualization and management of care teams within and across health care organizations. Electronic health information exchange has the potential to address this issue. In the present study, we assessed the adequacy of available health information exchange data standards to support the information needs related to care coordination of complex pediatric patients. METHODS: We derived data elements from the information needs of clinicians and parents to support patient care teams; and mapped them to data elements in the Health Level Seven (HL7) Consolidated Clinical Document Architecture (C-CDA) standard and in the HL7 Fast Healthcare Interoperability Resources (FHIR) standard. We also identified additional C-CDA data elements and FHIR resources that include patients' care team members. RESULTS: Information about care team members involved in patient care is generally well-represented in the C-CDA and FHIR specifications. However, there are gaps related to patients' non-clinical events and care team actions. In addition, there is no single place to find information about care team members; rather, information about practitioners and organizations may be available in several different types of C-CDA data elements and FHIR resources. CONCLUSION: Through standards-based electronic health information exchange, it appears to be feasible to build patient care team representations irrespective of the location of patient care. In order to gather care team information across disparate systems, exchange of multiple C-CDA documents and/or execution of multiple FHIR queries will be necessary. This approach has the potential to enable comprehensive patient care team views that may help improve care coordination.

Comprehensive methodology to monitor longitudinal change patterns during EHR implementations: a case study at a large health care delivery network.

OBJECTIVE: To test a systematic methodology to monitor longitudinal change patterns on quality, productivity, and safety outcomes during a large-scale commercial Electronic Health Record (EHR) implementation. MATERIALS AND METHODS: Our method combines an interrupted time-series design with control sites and 41 consensus outcomes including quality (11 measures), productivity (20 measures), and safety (10 measures). The intervention consisted of a phased commercial EHR implementation at a large health care delivery network. Four medium-size hospitals and 39 clinics from 5 geographic regions implementing the new EHR were compared against a parallel control consisting of one medium-size and one large hospital and 10 clinics that had not implemented the new EHR at the time of this study. We collected monthly data from February 2013 to July 2017. RESULTS: The proposed methodology was successfully implemented and significant changes were observed in most measured variables. A significant change attributable to the intervention was observed in 12 (29%) measures in three or more regions; in 32 (78%) measures in two or more regions; and in 40 (98%) measures in at least one region. A similar pattern (i.e., same impact in three or more regions) was detected for nine (22%) measures, a mixed pattern (i.e., same impact in two regions, and different impact in other regions) was detected for nine (22%) measures, and an inconsistent pattern (i.e., did not detect the same impact across regions) was detected for 23 (56%) measures. DISCUSSION: Using a formal methodology to assess changes in a set of consensus measures, we detected various patterns of impact and mixed time-sensitive effects. With an increasing adoption of EHR systems, it is critical for health care organizations to systematically monitor their EHR implementations. The proposed method provides a robust and consistent approach to monitor EHR implementations longitudinally allowing for continuous monitoring after the system becomes stable in order to avoid unexpected effects. CONCLUSION: Our results and methodology can guide the broader medical and informatics communities by informing what and how to continuously monitor EHR impact on quality, productivity, and safety.

Electronic problem lists: a thematic analysis of a systematic literature review to identify aspects critical to success.

Objective: Problem list data is a driving force for many beneficial clinical tools, yet these data remain underutilized. We performed a systematic literature review, pulling insights from previous research, aggregating insights into themes, and distilling themes into actionable advice. We sought to learn what changes we could make to existing applications, to the clinical workflow, and to clinicians' perceptions that would improve problem list utilization and increase the prevalence of problems data in the electronic medical record. Materials and Methods: We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to systematically curate a corpus of pertinent articles. We performed a thematic analysis, looking for interesting excerpts and ideas. By aggregating excerpts from many authors, we gained broader, more inclusive insights into what makes a good problem list and what factors are conducive to its success. Results: Analysis led to a list of 7 benefits of using the problem list, 15 aspects critical to problem list success, and knowledge to help inform policy development, such as consensus on what belongs on the problem list, who should maintain the problem list, and when. Conclusions: A list of suggestions is made on ways in which the problem list can be improved to increase utilization by clinicians. There is also a need for standard measurements of the problem list, so that lists can be measured, compared, and discussed with rigor and a common vocabulary.

Promoting national and international standards to build interoperable clinical applications.

During the last decade, software supporting healthcare delivery has proliferated. This software can be divided into electronic medical record (EHR) systems and applications that treat EHRs as platforms. These collect, manage, and interpret medical data, thereby adding value to associated EHRs. To reduce the burden of developing for multiple EHR platforms, a group of standards has evolved that allow software written for one vendor's EHR to be introduced into settings supported by other vendors. The Health Services Platform Consortium (HSPC) is a collaborative effort to advocate for standards that will make healthcare applications truly interoperable. In this document, we discuss the approach adopted by the consortium and the standards central to this approach. We discriminate between interoperability standards that support the plug-and-play transfer of applications from one vendor's EHR to another and knowledge portability standards that allow knowledge artifacts used in one software environment to be introduced effectively in others.

Enhancing a Commercial EMR with an Open, Standards-Based Publish-Subscribe Infrastructure.

Intermountain Healthcare has designed and implemented a publish-subscribe (PubSub) infrastructure to support essential event processing workflows across our organization. A recent implementation of a commercial EMR highlighted the need to provide this capability on top of the EMR to support external applications and services that require access to triggering events within the EMR. A description of the PubSub architecture is presented. Use cases for health information exchange, public health reporting, and pulmonary embolism diagnosis that utilize PubSub are described, along with benefits of using the paradigm. Besides providing support for these external applications, the PubSub infrastructure allows additional event handling functionality not available in the commercial EMR. The open, standards-based nature of the design should allow other organizations to implement the system in their information systems environment.

Development and classification of a robust inventory of near real-time outcome measurements for assessing information technology interventions in health care.

OBJECTIVE: To develop and classify an inventory of near real-time outcome measures for assessing information technology (IT) interventions in health care and assess their relevance as perceived by experts in the field. MATERIALS AND METHODS: To verify the robustness and coverage of a previously published inventory of measures and taxonomy, we conducted semi-structured interviews with clinical and administrative leaders from a large care delivery system to collect suggestions of outcome measures that can be calculated with data available in electronic format for near real-time monitoring of EHR implementations. We combined these measures with the most commonly reported in the literature. We then conducted two online surveys with subject-matter experts to collect their perceptions of the relevance of the measures, and identify other potentially relevant measures. RESULTS: With input from experienced health care leaders and informaticists, we developed an inventory of 102 outcome measures. These measures were classified into a taxonomy of commonly used measures around the categories of quality, productivity, and safety. Safety measures were rated as most relevant by subject-matter experts, especially those measuring medication processes. Clinician satisfaction and measures assessing mean time to complete tasks and time spent on electronic documentation were also rated as highly relevant. DISCUSSION: By expanding the coverage of our previously published inventory and taxonomy, we expect to help providers, health IT vendors and researchers to more effectively and consistently monitor the impact of EHR implementations in near real-time, and report more standardized outcomes in future studies. We identified several measures not commonly assessed by previous studies of IT implementations, especially those of safety and productivity, which deserve more attention from the broader informatics community. CONCLUSION: Our inventory of measures and taxonomy will help researchers identify gaps in their measurement approaches and report more standardized measurements of IT interventions that could be shared among researchers, hopefully facilitating comparison across future studies and increasing our understanding of the impact of IT interventions in health care.

Evaluation of a systematic methodology to detect in near real-time performance changes during electronic health record system implementations: a longitudinal study.

Introduction. Although Electronic Health Record (EHR) adoption has increased in the U.S., our understanding of how it affects health care organizations is still limited. Current literature has produced mixed-results due to the use of simple, non-standardized measurements and poor research designs. Methods. We propose the use of a systematic methodology that combines measures of quality, productivity and safety processes, tracked over time using an interrupted time-series design with multiple control sites. Results. Our methodology successfully detected performance changes during an EHR implementation on 17 (77%) outcomes, including a significant increase in Emergency Department length of stay immediately after go live by 0.19 hours [95%CI (0.12, 0.27), p<0.001], and an improvement in time to complete radiology tests, which significantly decreased per month by 0.19 minutes [95%CI (-0.26, -0.12), p<0.001]. Conclusion. The proposed methodology was able to detect several changes immediately after an EHR implementation and over time. The method is a promising and robust approach to assessing the impact of EHR implementations on a wide range of health care quality, productivity, and safety care processes.

Health information technology adoption: Understanding research protocols and outcome measurements for IT interventions in health care.

OBJECTIVE: To classify and characterize the variables commonly used to measure the impact of Information Technology (IT) adoption in health care, as well as settings and IT interventions tested, and to guide future research. MATERIALS AND METHODS: We conducted a descriptive study screening a sample of 236 studies from a previous systematic review to identify outcome measures used and the availability of data to calculate these measures. We also developed a taxonomy of commonly used measures and explored setting characteristics and IT interventions. RESULTS: Clinical decision support is the most common intervention tested, primarily in non-hospital-based clinics and large academic hospitals. We identified 15 taxa representing the 79 most commonly used measures. Quality of care was the most common category of these measurements with 62 instances, followed by productivity (11 instances) and patient safety (6 instances). Measures used varied according to type of setting, IT intervention and targeted population. DISCUSSION: This study provides an inventory and a taxonomy of commonly used measures that will help researchers select measures in future studies as well as identify gaps in their measurement approaches. The classification of the other protocol components such as settings and interventions will also help researchers identify underexplored areas of research on the impact of IT interventions in health care. CONCLUSION: A more robust and standardized measurement system and more detailed descriptions of interventions and settings are necessary to enable comparison between studies and a better understanding of the impact of IT adoption in health care settings.

Building an Ontology for Identity Resolution in Healthcare and Public Health.

UNLABELLED: Integration of disparate information from electronic health records, clinical data warehouses, birth certificate registries and other public health information systems offers great potential for clinical care, public health practice, and research. Such integration, however, depends on correctly matching patient-specific records using demographic identifiers. Without standards for these identifiers, record linkage is complicated by issues of structural and semantic heterogeneity. OBJECTIVES: Our objectives were to develop and validate an ontology to: 1) identify components of identity and events subsequent to birth that result in creation, change, or sharing of identity information; 2) develop an ontology to facilitate data integration from multiple healthcare and public health sources; and 3) validate the ontology's ability to model identity-changing events over time. METHODS: We interviewed domain experts in area hospitals and public health programs and developed process models describing the creation and transmission of identity information among various organizations for activities subsequent to a birth event. We searched for existing relevant ontologies. We validated the content of our ontology with simulated identity information conforming to scenarios identified in our process models. RESULTS: We chose the Simple Event Model (SEM) to describe events in early childhood and integrated the Clinical Element Model (CEM) for demographic information. We demonstrated the ability of the combined SEM-CEM ontology to model identity events over time. CONCLUSION: The use of an ontology can overcome issues of semantic and syntactic heterogeneity to facilitate record linkage.

Birth of identity: understanding changes to birth certificates and their value for identity resolution.

INTRODUCTION: Identity information is often used to link records within or among information systems in public health and clinical settings. The quality and stability of birth certificate identifiers impacts both the success of linkage efforts and the value of birth certificate registries for identity resolution. OBJECTIVE: Our objectives were to describe: (1) the frequency and cause of changes to birth certificate identifiers as children age, and (2) the frequency of events (ie, adoptions, paternities, amendments) that may trigger changes and their impact on names. METHODS: We obtained two de-identified datasets from the Utah birth certificate registry: (1) change history from 2000 to 2012, and (2) occurrences for adoptions, paternities, and amendments among births in 1987 and 2000. We conducted cohort analyses for births in 1987 and 2000, examining the number, reason, and extent of changes over time. We conducted cross-sectional analyses to assess the patterns of changes between 2000 and 2012. RESULTS: In a cohort of 48 350 individuals born in 2000 in Utah, 3164 (6.5%) experienced a change in identifiers prior to their 13th birthday, with most changes occurring before 2 years of age. Cross-sectional analysis showed that identifiers are stable for individuals over 5 years of age, but patterns of changes fluctuate considerably over time, potentially due to policy and social factors. CONCLUSIONS: Identities represented in birth certificates change over time. Specific events that cause changes to birth certificates also fluctuate over time. Understanding these changes can help in the development of automated strategies to improve identity resolution.

Longitudinal Analysis of Computerized Alerts for Laboratory Monitoring of Post-liver Transplant Immunosuppressive Care.

Post-liver transplant patients require lifelong immunosuppressive care and monitoring. Computerized alerts can aid laboratory monitoring, but it is unknown how the distribution of alerts changes over time. We describe the changes over time of the distribution of computerized alerts for laboratory monitoring of post-liver transplant immunosuppressive care. Data were collected for post-liver transplant patients transplanted and managed at Intermountain Healthcare between 2005 and 2012. Alerts were analyzed based on year triggered, time since transplantation, hospitalization status, alert type, action taken (accepted or rejected), reason given for the action taken, and narrative comments. Alerts for overdue laboratory testing became more prevalent as time since transplantation increased. There is an increased need to support monitoring for overdue laboratory testing as the time since transplantation increases. Alerts should support providers as they monitor the evolving needs of post-transplant patients over time. We identify opportunities for improving laboratory monitoring of post-liver transplant patients.

A domain analysis model for eIRB systems: addressing the weak link in clinical research informatics.

Institutional Review Boards (IRBs) are a critical component of clinical research and can become a significant bottleneck due to the dramatic increase, in both volume and complexity of clinical research. Despite the interest in developing clinical research informatics (CRI) systems and supporting data standards to increase clinical research efficiency and interoperability, informatics research in the IRB domain has not attracted much attention in the scientific community. The lack of standardized and structured application forms across different IRBs causes inefficient and inconsistent proposal reviews and cumbersome workflows. These issues are even more prominent in multi-institutional clinical research that is rapidly becoming the norm. This paper proposes and evaluates a domain analysis model for electronic IRB (eIRB) systems, paving the way for streamlined clinical research workflow via integration with other CRI systems and improved IRB application throughput via computer-assisted decision support.

Problem management module: an innovative system to improve problem list workflow.

Electronic problem lists are essential to modern health record systems, with a primary goal to serve as the repository of a patient's current health issues. Additionally, coded problems can be used to drive downstream activities such as decision support, evidence-based medicine, billing, and cohort generation for research. Meaningful Use also requires use of a coded problem list. Over the course of three years, Intermountain Healthcare developed a problem management module (PMM) that provided innovative functionality to improve clinical workflow and boost problem list adoption, e.g. smart search, user customizable views, problem evolution, and problem timelines. In 23 months of clinical use, clinicians entered over 70,000 health issues, the percentage of free-text items dropped to 1.2%, completeness of problem list items increased by 14%, and more collaborative habits were initiated.

A focus area maturity model for a statewide master person index.

OBJECTIVE: The sharing of personally identifiable information across organizational boundaries to facilitate patient identification in Utah presents significant policy challenges. Our objective was to create a focus area maturity model to describe and evaluate our progress in developing a policy framework to support a statewide master person index (sMPI) for healthcare and public health operations and research in Utah. MATERIALS AND METHODS: We used various artifacts, including minutes from policy guidance committee meetings over a span of 18 months, a report from Utah's Digital Health Services Commission, and a draft technical requirements document to retrospectively analyze our work and create a focus area maturity model describing the domain of policy needed to support the sMPI. We then used our model to assess our progress and future goals. CONCLUSIONS: The focus area maturity model provides an orderly path that can guide the complex process of developing a functional statewide master person index among diverse, autonomous partners. While this paper focuses on our experience in Utah, we believe that the arguments for using a focus area maturity model to guide the development of state or regional MPIs is of general interest.

Assessing clinical researchers' information needs to create responsive portals and tools: my Research Assistant (MyRA) at the University of Utah: a case study.

QUESTION: How can health sciences librarians and biomedical informaticians offer relevant support to Clinical and Translational Science Award (CTSA) personnel? SETTING: The Spencer S. Eccles Health Sciences Library and the associate vice president for information technology for the health sciences office at the University of Utah conducted a needs assessment. METHODS: Faculty and staff from these two units, with the services of a consultant and other CTSA partners, employed a survey, focus groups, interviews, and committee discussions. An information portal was created to meet identified needs. RESULTS: A directive white paper was created. The process employed to plan a virtual and physical collaborative, collegial space for clinical researchers at the university and its three inter-institutional CTSA partners is described. CONCLUSION: The university's model can assist other librarians and informaticians with how to become part of a CTSA-focused infrastructure for clinical and translational research and serve researchers in general.